Installing Python Packages
As a part of a Tier One public research institution, the HPE Data Science Institute (HPE DSI) offers three high-performance supercomputing clusters (Opuntia, Sabine, and Carya) to researchers within the University of Houston. However, when it comes to installing Python packages on these clusters, some may run into job-specific errors or memory issues. Ebalunode’s workshop served as an in-depth guide for those looking to mitigate these problems, discussing the specifics of utilizing virtual environments (VEs) and PIP in these settings.
In the first half of the workshop, Ebalunode focused on virtual environments, which are directories that contain a specific collection of Python packages or libraries. These applications can be easily switched, according to the workload and type of Python version a user wants. He taught participants how to create separate environments using the venv command and use modules on them. Ebalunode advised researchers to install VEs on their own project directory, and to utilize installation software, like a SLURM interactive job session, when dealing with larger installations.
In the second half, Ebalunode briefly discussed the installation of packages on PIP, the standard Python package manager. They can be used alongside virtual environments and are best in place of a Conda package (or for updating existing ones.)
To demonstrate in further detail, Ebalunode walked through the basic steps of installation and activation for each of the two tools. Using the sci-kit learn library as an example, he explained how to activate a virtual environment and use it inside a SLURM batch job, as well as showed how to run PIP to install a package to a specific path.
To access the HPE DSI platforms, UH faculty and researchers can request cluster access online. For more information about RCDC resources and information, please visit the RCDC website.